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Fielding  operationally  suitable  weapon  systems  is  a  prime 
objective  of  Defense  acquisition.  An  operationally  suitable 
weapon  system  is  one  that  is  available  for  combat  when 
needed,  is  reliable  enough  to  accomplish  the  mission, 
operates  satisfactorily  with  Service  personnel  and  other 
systems,  and  does  not  impose  an  undue  logistics  burden  in 
peacetime  or  wartime.  Operational  test  and  evaluation 
(OT&E)  is  required  to  evaluate  the  effectiveness  and 
suitability  of  major  systems  before  the  full-rate  production 
decision. 

Our  review  of  seven  systems  fielded  over  the  past  6  years 
compared  OT&E  results  with  field  experience.  This  review 
showed  both  strengths  and  weaknesses  in  the  treatment  of 
suitability.  On  balance,  OT&E  has  been  a  reasonable 
predictor  of  suitability.  However,  several  systems  have  not 
achieved  their  availability  goals,  and  they  consume 
significantly  more  logistics  resources  than  anticipated.  We 
believe  that  many  of  the  major  suitability  problems  could 
have  been  avoided  through  better  suitability  assessment. 
We  present  findings  from  the  case  studies,  conclusions 
about  the  causes  of  the  problems,  and  recommendations  for 
improving  suitability  assessment. 

S  UITABILIT  Y  ASSESSMENT  FINDINGS 

To  interpret  the  case  studies,  we  investigated  the  three 
elements  of  a  weapon  system:  the  end  item  (or  primary 
mission  equipment),  peculiar  support  (e.g.,  test  program 
sets  and  training),  and  general  logistics  support  (e.g.,  supply 
support). 

OT&E  found  most  major  problems  that  relate  to  end-item 
suitability.  For  example,  reliability  problems  were  detected 
in  environmental  control  equipment,  propulsion  equipment, 
avionics,  and  software.  However,  OT&E  failed  to  identify 


Executive  Summary 


Page  iii 


some  suitability  problems  associated  with  cumulative 
effects,  such  as  corrosion  and  fatigue. 

Peculiar  support  was  only  partially  tested  during  OT&E. 
As  a  result,  OT&E  did  not  find  problems  that  existed  in  a 
major  fault  diagnosis  system  and  in  some  avionics  support. 
When  peculiar  support  was  thoroughly  tested,  problems 
were  found  and  corrected  before  fielding. 

General  logistics  support  was  not  tested  and  was  seldom 
evaluated  during  OT&E.  General  logistics  support 
elements  were  not  included  in  operational  tests  of  the  case 
studies.  In  two  cases,  field  suitability  has  been  notably 
degraded  by  supply  problems. 

Three  additional  general  findings  are  noteworthy.  First, 
OT&E  did  not  raise  any  false  alarms.  The  problems  cited  by 
OT&E  suitability  evaluations  were  real  problems.  Second, 
OT&E  reports  focused  on  the  items  tested  and  the  data  from 
the  tests.  Little  use  was  made  of  data  from  other  sources, 
such  as  development  testing  or  logistics  support  analysis,  or 
modeling  and  analysis  to  augment  the  suitability  test  data. 
Third,  corrective  action  programs  can  be  used  prior  to 
fielding  to  correct  suitability  problems  found  in  OT&E. 
Significant  problems  found  in  OT&E  were  corrected 
through  dedicated  programs  that  specified  actions  and  those 
responsible  for  the  actions. 

CONCLUSIONS 

Acquisition  policy  to  ensure  the  suitability  of  fielded 
systems  is  in  place.  The  shortcomings  in  OT&E  could  not  be 
attributed  to  inadequate  policy.  The  5000  series  of  DoD 
instructions  provides  the  direction  and  defines  the  products 
needed  at  each  milestone.  This  policy  recognizes  the 
iterative  nature  of  suitability  testing,  and  that  suitability 
assessment  is  the  responsibility  of  the  entire  test  and 
evaluation  (T&E)  community.  Because  suitability  problems 
are  reaching  the  field  without  being  identified  (even  though 
the  policy  is  adequate),  it  follows  that  policy 
implementation  needs  to  be  improved. 

The  crucial  suitability  issues  are  not  adequately  identified 
early  in  the  T&E  process.  For  the  case  studies,  this  was  the 
primary  reason  why  the  suitability  assessments  of  the 
fielded  systems  revealed  problems  not  identified  in  OT&E. 
An  issue  may  be  crucial  because  of  risks  associated  with 
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new  technologies,  operational  concepts,  or  support  concepts, 
or  because  it  represents  a  major  consumer  of  logistics 
resources.  If  the  crucial  suitability  issues  are  not  identified 
early  and  addressed  in  test  plans,  then  operational  tests  will 
not  reveal  problems  associated  with  them. 

Inadequate  treatment  of  peculiar  support  equipment  is  a 
serious  shortcoming.  Support  equipment  is  frequently 
omitted  from  test  plans  and  not  adequately  considered  in 
operational  suitability  assessments.  Both  wartime 
performance  and  peacetime  cost  of  a  system  can  depend 
heavily  on  peculiar  support  equipment.  Increasing  use  of 
electronics  and  two-level  maintenance  will  exacerbate  the 
issue. 

RECOMMENDA  TIONS 

To  better  put  current  suitability  assessment  policy  into 
practice  at  the  Office  of  the  Secretary  of  Defense  (OSD)  level 
we  recommend  that: 

•  Increased  attention  and  technical  support  be 
devoted  to  suitability  assessment  and  testing  at 
each  phase  of  development  to  help  find  deficiencies 
as  early  as  possible 

•  Increased  use  of  corrective  action  programs  be 
made  in  the  most  appropriate  phase  to  correct 
deficiencies 

•  Results  and  data  from  all  phases  and  technical 
disciplines  be  made  available  and  used  to  better 
coordinate  and  integrate  suitability  test  and 
evaluation  throughout  development 

•  Critical  items  of  peculiar  support  equipment  be 
identified,  included  in  the  test  plans,  and  made 
available  for  testing. 

We  make  specific  suggestions  on  how  these 
recommendations  can  be  applied  by  OT&E  action  officers. 
These  suggestions  should  be  extended  across  the  board  to  all 
OSD  offices  concerned  with  suitability  assessment.  Doing 
so  will  help  ensure  that  suitability  issues  are  considered  and 
that  proper  actions  are  taken  to  get  more  suitable  systems 
into  the  field.  Judging  from  past  experience,  such  an  effort 
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is  needed  and  will  pay  for  itself  many  times  over  in 
improved  availability  and  avoided  logistics  costs. 
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CHAPTER  1.  NATURE  OF  THE  PROBLEM 


Fielding  operationally  suitable  systems  is  a  prime  objective 
of  Defense  acquisition.  A  suitable  weapon  system  is  one 
that  is  available  for  combat  when  needed,  is  reliable  enough 
to  accomplish  the  mission,  operates  satisfactorily  with 
Service  personnel  and  other  systems,  and  does  not  impose 
an  undue  logistics  burden  in  peacetime  or  wartime.  1 
Concern  about  the  Department’s  success  in  fielding  suitable 
systems  was  expressed  in  an  October  1990  memorandum 
from  the  DUSD(A). 

The  Director,  Operational  Test  and  Evaluation  (DOT&E),  is 
required  by  Title  10,  Sec.  2399,  United  States  Code  (U.S.C.)l 2 
to  report  on  the  operational  effectiveness  and  suitability  of 
major  systems  before  the  full-rate  production  decision.  His 
report  is  to  be  based  on  an  evaluation  of  whether  the  test 
and  evaluation  (T&E)  performed  were  adequate  and 
whether  the  results  confirm  that  the  items  or  components 
actually  tested  are  effective  and  suitable  for  combat.  As  a 
"final”  check  on  suitability  before  production,  DOT&E  helps 
assure  that  systems  will  be  suitable  when  fielded. 

RESULTS  OF  CASE  STUDIES 

To  determine  whether  systems  were  suitable  when  fielded 
and  what  differences  exist  between  suitability  as  observed 
in  operational  test  and  evaluation  (OT&E)  and  suitability 
as  observed  in  the  field,  we  examined  seven  cases.  The  case 
studies  are  listed  in  Table  1-1.  They  were  chosen  to 
represent  each  Service  and  each  major  conventional  warfare 
area.  In  addition,  each  system  had  to  have  gone  through 
OT&E  relatively  recently  and  yet  have  accumulated 
sufficient  field  experience  for  suitability  judgments  to  be 
made.  Since  the  cases  were  not  intended  to  be  a  statistically 
significant  or  representative  sample,  no  extrapolations 
should  be  made  from  them  to  determine  the  percentage  of 

lThe  official  definition  of  operational  suitability  is  the  degree  to  which  a  system  can  be  placed 
satisfactorily  in  field  use  with  consideration  given  to  availability,  compatibility,  transportability, 
interoperability,  reliability,  wartime  usage  rates,  maintainability,  safety,  human  factors,  manpower 
supportability,  logistics  supportability,  documentation,  and  training  requirements. 

2The  complete  text  of  U  S.  Code,  Title  10,  Section  2399  is  in  the  Appendix. 
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unsuitable  systems  or  the  total  cost  of  correcting  unsuitable 
systems,  or  to  derive  any  other  statistic  of  that  nature.  The 
cases  did,  however,  exhibit  a  range  of  outcomes,  and  they  do 
illustrate  certain  problems  and  opportunities  for 
improvements  in  the  suitability  assessment  process. 

TABLE  1-1 
CASE  STUDIES 


System 

Service 

Date  of  OT&E 

Mobile  subscriber  equipment  (MSE) 

Army 

Aug  -  Oct.  1988 

M939-A2  5-ton  truck 

Army 

Aug. -Oct.  1988 

AH-64  Apache  helicopter 

Army 

Jut.  -  Sep.  1986 

Landing  craft  air  cushion  (LCAC) 

Navy 

Feb.  1985 -Apr.  1987 

AV-8B  Harrier 

Navy/Marine  Corps 

Aug  1984 -Mar.  1985 

LANTIRN  NAV  Pod 

Air  Force 

Apr.  -  Nov  1989 

F-16GD  Block  40 

Air  Force 

May -Oct.  1989 

Specific  System  Findings 

The  suitability  of  the  systems  when  tested  and  when  fielded 
varied  widely  across  the  seven  case  studies. 

The  mobile  subscriber  equipment  (MSE)  and  the  5-ton  truck 
both  exhibited  problems  in  OT&E  that  were  corrected  by 
aggressive  corrective  action  programs,  and  both  have  since 
proven  to  be  suitable  in  the  field. 

The  AH-64,  although  rated  as  potentially  suitable  at  the 
end  of  OT&E,  has  failed  to  meet  its  suitability  goals.  Its 
availability,  as  measured  by  fully  mission  capable  rate,  has 
been  50  to  65  percent  as  opposed  to  the  goal  of  70  percent. 
That  means  the  downtime  has  been  17  to  67  percent  more 
than  anticipated.  This  is  true  even  though  more  money 
than  anticipated  was  spent  for  spare  parts  to  compensate  for 
reliability  problems  in  major  components  such  as  the  rotor 
blades  and  the  shaft-driven  compressor.  The  cost  of  these 
extra  parts  is  conservatively  estimated  at  $200  million.  In 
our  judgment,  these  and  other  suitability  problems  could 
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have  been  detected  and  flagged  in  operational  testing  (OT) 
and/or  development  testing  (DT). 

The  fielding  of  the  Landing  Craft  Air  Cushion  (LCAC)  was 
delayed  as  a  result  of  reliability  prob’ems  found  during 
OT&E.  Eventually,  these  problems  were  corrected  and  the 
system  was  judged  potentially  suitable.  However,  after 
fielding,  it  still  experienced  suitability  problems  in  the 
areas  of  corrosion,  training,  and  supply  support  that  could 
and  should  have  been  identified.  To  illustrate  the  financial 
cost  of  not  identifying  such  problems  early,  the  life-cycle 
cost  to  correct  LCAC  corrosion  problems  that  could  have 
been  avoided  totals  over  $170  million. 

The  AV-8B  was  judged  potentially  suitable.  In  peacetime, 
the  AV-8B  has  not  reached  its  mission-capable  targets 
because  of  lower-than-planned  levels  of  maintenance 
manning  and  spares  support.  When  these  resources  were 
raised  to  the  planned  levels  during  Desert  Storm,  the 
aircraft  was  suitable. 

The  LANTIRN  NAV  Pod  progressed  from  “not  opera¬ 
tionally  suitable”  in  its  first  OT&E  to  “operationally 
suitable”  in  follow-on  OT&E  (FOT&E).  Observed  reliability 
growth  has  been  slower  than  predicted,  but  the  NAV  pod 
reliability  eventually  exceeded  its  reliability  goal.  The 
engineering  change  process  behind  the  improvement  was  an 
informal  corrective  action  program.  Field  experience  shows 
that  the  NAV  pod  is  indeed  suitable 

The  F-16  C/D  Block  40  met  almost  all  of  its  suitability  goals 
at  OT&E,  although  it  had  several  avionics  reliability 
problems  that  have  not  yet  been  fully  solved.  The  aircraft  is 
suitable  in  the  field,  consistently  exceeding  its  availability 
goal. 

Volume  II  of  this  report  contains  the  detailed  case  studies 
covering  each  of  the  seven  systems  just  described. 

General  OT&E  Findings 

Three  general  findings  are  noteworthy.  First,  OT&E  did 
not  raise  any  false  alarms.  In  the  case  studies,  all  of  the 
problems  noted  in  OT&E  were  real  problems.  If  they  were 
not  resolved  prior  to  fielding,  they  showed  up  after  fielding 

The  second  general  finding  from  the  cases  studied  is  that 
OT&E  reports  focused  on  what  was  tested  and  on  data 
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flowing  from  that  test.  The  reports  did  not  use  data  from 
other  sources  or  (with  rare  exceptions)  use  modeling  and 
analysis  to  assist  in  evaluation.  (We  speculate  that  this 
situation  resulted  in  part  from  the  congressional  language 
and  concerns  reflected  in  Title  10,  Sec.  2399,  U.S.C.  This 
language  [see  the  Appendix]  discourages  the  use  of 
modeling  in  lieu  of  test  and  speaks  of  the  suitability  of  the 
item  tested,  not  the  suitability  of  the  complete  system  to  be 
fielded.  Its  intent  is  to  prevent  suitability  judgments  from 
being  made  on  promised  rather  than  actual  performance. 
However,  it  should  not  be  extended,  in  our  judgment,  to 
preclude  the  use  of  all  other  sources  of  data  or  of  modeling  to 
augment  test  data  and  help  evaluate  the  results.  OT&E  is 
specifically  authorized  by  law  to  have  access  to  such 
information.) 

The  third  general  finding  is  that  corrective  action 
programs  (CAPs)  can  be  used  successfully  prior  to 
fielding  to  correct  suitability  problems  found  in  OT&E. 

The  MSE  and  the  5-ton  truck  are  good  examples.  In  the  case 
of  the  MSE,  significant  problems  were  found  in  OT  and  were 
corrected  in  a  short  time  through  a  dedicated  program  that 
laid  out  specific  actions  and  those  responsible  for  taking 
those  actions.  The  improvements  resulting  from  the 
program  were  tested  in  turn  in  subsequent  FOT&E.  In  the 
case  of  the  5-ton  truck,  the  corrective  action  program  was 
undertaken  while  the  test  was  in  progress  and  succeeded  by 
the  end  of  the  test  in  correcting  the  relatively  minor 
problems  found  in  the  test. 

Analysis  of  Findings  —  Elements  of  a 
Suitable  System 

To  interpret  the  case  study  findings,  we  first  briefly  review 
the  three  elements3  of  a  suitable  system.  We  then  discuss 
the  findings  for  each  element. 

The  first  element  is  the  "end  item”  itself.  At  the  time  when 
OT&E  takes  place,  just  before  the  full-rate  production 
decision,  the  end  item  under  test  should  be  a  "production 
representative”  article.  It  is  usually  a  low-rate  initial 


^These  three  elements  are  equivalent  to  those  defined  in  DoD  Instruction  5000.2  as  “the  prime 
mission  equipment,"  “the  logistics  support  structure  for  the  system,”  and  "the  other  elements  of  the 
operational  support  infrastructure  within  which  the  system  must  operate  ” 
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production  (or  pilot  production)  item.  Thus  the  end-item 
configuration  at  this  point  should  be  relatively  stable. 

The  second  element  of  the  system  is  its  “peculiar  support.” 
At  OT&E,  this  element  -  which  includes  built-in  test  (BIT) 
and  external  test  equipment,  technical  data,  training 
curricula  and  materials,  facilities,  and  unique  depot  repair 
capabilities  -  is  often  less  mature  than  the  end  item.  In 
fact,  some  parts  of  the  peculiar  support  may  not  be  available 
for  testing  at  this  stage. 

The  third  and  final  element  of  system  suitability  is  the  total 
Service  logistic  system  within  which  the  weapon  system 
operates  -  called  here  “general  logistics  support.”  At 
OT&E,  this  infrastructure  is  usually  not  represented  at  all. 
In  some  cases  it  is  simulated  (for  example,  during  the  test 
the  resupply  of  spare  parts  may  be  delayed  for  a  time  that  is 
thought  to  be  representative  of  the  delays  within  the 
Service  logistics  system).  After  fielding,  however,  it  is  this 
support  infrastructure  that  will  provide  the  supplies  and 
repairs  that  the  system  will  need  in  order  to  operate.  Thus, 
the  support  infrastructure  often  has  a  critical  influence  on 
the  observed  suitability  of  the  system  in  the  field. 

Table  1-2  summarizes  the  most  significant  findings  from  the 
case  studies. 


TABLE  1-2 

RESULTS  OF  OT&E  VERSUS  FIELD  SUITABILITY 


Suitability  element 

Finding 

End  item 

Peculiar  support 

General  logistics  support 

OT&E  found  most  major  problems;  missed  cumulative 
effects  problems. 

Problems  missed;  often  not  tested. 

Generally  not  tested  or  evaluated. 

End  Items 

OT&E  found  most  major  problems  relating  to  end-item 
suitability.  For  example,  reliability  problems  such  as  the 
air  conditioner  on  the  AV-8B,  software  in  the  MSE,  and 
avionics  in  the  F-16  were  detected.  The  one  class  of  end 
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item  problems  not  found  were  those  relating  to  cumulative 
effects.  Some  examples  are  delamination  of  AH-64  main 
rotor  blades,  heat-shield  wearout  on  the  AV-8B,  corrosion 
problems  on  the  LCAC,  and  generator  aging  problems  on 
the  MSE.  The  OT  did  not  last  long  enough  for  these 
problems  to  show  up  during  the  test,  and  no  attempt  was 
made  in  the  evaluation  of  test  results  to  use  DT  results  or 
other  information  to  predict  what  suitability  problems 
might  appear  after  extended  use  in  the  field. 


Cumulative  effects  example:  Heat  shield  wearout  on  the  AV-8B 

The  AV-8B  uses  two  titanium  heat  shields  to  deflect  hot  exhaust  from  the  aircraft  structure. 
The  heat  shields  are  cited  by  maintenance  personnel  in  the  Harrier  squadrons  as  one  of  the  chief 
maintenance  burdens.  The  heat  shields  were  designed  to  be  removed  and  refurbished  when 
worn.  However,  removal  requires  drilling  out  the  rivets  holding  the  shield  to  the  structure  and 
then  using  larger  rivets  to  reattach  the  shield.  After  this  procedure  has  been  done  several  times, 
the  holes  in  the  heat  shield  that  have  been  drilled  out  have  to  be  built  back  up  to  the  original 
specification.  This  process  requires  a  higher  level  of  maintenance  and  is  time  consuming.  The 
problem  was  not  recognized  in  OT  because  the  procedure  did  not  have  to  be  carried  out 
repetitively  in  the  limited  time  OT  took  place. 


Peculiar  Support 

Peculiar  support  was  only  partially  tested.  But  even  in 
some  cases  (such  as  LANTIRN)  in  which  peculiar  support 
was  only  partially  tested,  the  »ysicm  proved  to  be  suitable; 
others,  such  as  the  Fault  Detection/Location  System 
(FD/LS)  on  the  AH-64,  and.  to  a  lesser  extent,  F-16  Block  40 
avionics,  had  unanticipated  problems  later  in  fielding. 
Where  peculiar  support  was  thoroughly  tested  (for  example, 
new  equipment  training  in  the  MSE),  problems  were  found 
and  subsequently  corrected  before  fielding. 

This  inconsistent  treatment  of  peculiar  support  is  in  our 
opinion  a  serious  shortcoming.  Both  the  wartime 
performance  and  peacetime  cost  of  weapon  systems  depend 
on  the  success  of  peculiar  support.  The  ability  to  accurately 
and  quickly  diagnose  faults,  verify  equipment  status,  and 
perform  calibrations  affects  availability,  mission  reliability, 
and  maintenance  and  training  requirements.  Critical  items 


Chapter  1.  Nature  of  the  Problem 


Page  7 


of  peculiar  support  should  be  identified  early  on,  made 
available  for  test,  and  thoroughly  tested. 

Test  designers  frequently  assume  that  support  equipment 
will  not  be  sufficiently  mature  for  testing  at  OT&E  and, 
therefore,  do  not  include  provisions  for  the  support 
equipment  in  the  test  plans.  Thus,  support  equipment 
performance  is  often  not  considered  in  the  OT&E 
operational  suitability  assessments. 

It  has  been  suggested  that,  in  some  cases,  peculiar  support 
equipment  could  have  been  made  available  for  testing  but 
was  not.  One  suggested  "rationale”  for  this  phenomenon 
was  that  if  the  peculiar  support  equipment  were  to  fail 
testing,  the  entire  system  acquisition  might  be  delayed. 

This  unavailability  of  critical  support  equipment  is  a 
widespread  problem.  It  highlights  the  need  for  discipline  in 
the  system  and  for  up-front  identification  of  peculiar 
support  that  is  crucial  to  suitability. 


Peculiar  support  example:  AH-64  Fault  Detection/Location  System  (FD/LS) 

The  FD/LS  provides  the  means  to  test  AH-64  subsystems  and  isolate  faults  to  the  line 
replaceable  unit  (LRU)  level.  It  is  designed  to  work  with  the  electronic  equipment  test  facility 
(EETF).  During  OT,  the  EETF  was  incomplete  and  could  not  be  assessed.  Hence,  the 
maintenance  concept  for  the  AH-64  could  not  be  tested.  Even  the  FD/LS  itself  was  not  tested 
thoroughly.  No  faults  were  inserted  to  test  its  diagnostic  powers  during  the  logistics 
demonstration,  and  although  inaccurate  fault  detection  was  noted  during  operational  test,  it  was 
not  judged  to  be  a  serious  problem.  Today,  FD/LS  still  has  a  40  percent  error  rate  in  fault 
detection. 


General  Logistics  Support 

General  logistics  support  was  not  tested  and  was 
seldom  evaluated  during  OT&E.  In  the  case  of  the 
LCAC,  wholesale  supply  support  later  proved  to  be  a  major 
field  suitability  problem.  The  AV-8B’s  availability  has  also 
suffered  from  supply  system  problems. 
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CONTEXT  OF  SUITABILITY  ASSESSMENT 

To  draw  conclusions  from  these  findings,  we  must  first  look 
beyond  OT&E  to  the  larger  context  of  suitability 
assessment.  Figure  1-1  pictures  the  various  phases  of 
suitability  testing.  Suitability  should  be  assessed  from 
early  development  through  fielding.  However,  not  all 
elements  of  suitability  will  be  demonstrable  at  all  times. 
On  one  hand,  for  instance,  the  reliability  of  the  prime 
mission  equipment  -  and  in  particular  the  reliability  of 
critical  components  -  may  be  demonstrated  relatively 
early  in  the  system’s  development.  On  the  other  hand,  the 
interaction  of  the  prime  mission  equipment  and  its  peculiar 
support  with  the  larger  logistics  system  may  be 
demonstrated  in  toto  only  after  the  system  is  fielded, 
perhaps  only  after  it  is  fielded  in  quantity.  Thus,  what  is 
available  to  be  tested  during  each  phase  differs. 


FIG.  1-1.  PHASES  OF  SUITABILITY  TESTING 

The  information  available  at  each  phase,  however,  does  not 
depend  solely  on  what  hardware  items  are  available  for  test. 
Results  of  previous  tests,  and  models  and  analyses,  can  be 
used  to  augment  test  data  and  help  project  future 
performance.  At  OT,  for  example,  the  results  of  DT  are 
known  and  can  often  be  used  to  inform  and  benefit  the  test 
and  evaluation  process.  Because  DT  is  continuous,  it  can 
use  results  of  earlier  tests  as  well.  Deciding  what  is 
essential  to  test  requires  using  models  and  analysis  to 
determine  what  will  be  crucial  suitability  determinants 
after  fielding.  Thus,  at  each  stage  of  testing,  looking  both 
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forward  to  fielding  and  backward  to  already  available 
information  is  essential  to  efficient  suitability  assessment. 

Findings  in  Context 

Looking  at  the  findings  from  the  OT&E  case  studies  in  this 
context,  the  greatest  difficulty  appears  to  result  from  the 
isolation  of  OT&E.  The  case  studies  show  that  OT  was 
successful  at  finding  the  end- item  suitability  problems  that 
were  present  in  the  specific  end  items  tested.  Otherwise, 
suitability  problems  were  often  not  found.  Problems  that 
could  have  been  apparent  from  DT  results  (such  as  corrosion 
on  the  LCAC)  were  not  investigated  in  DT  or  OT.  Elements 
that  would  become  crucial  to  meet  suitability  targets  when 
the  system  was  fielded  (such  as  the  fault  diagnostics  on  the 
AH-64)  were  not  rigorously  tested.  If  there  had  been  more 
interchange  with  the  DT  community  and  better 
understanding  of  the  crucial  suitability  issues,  more  of 
these  problems  might  have  been  identified  in  testing.  If 
such  issues  are  to  be  (a)  identified  as  critical  before  testing 
and  (b)  acted  on  as  a  result  of  the  tests,  then  the  Deputy 
Director  of  Defense  Research  and  Engineering  (Test  and 
Evaluation)  DDDRE(T&E)  and  DOT&E  will  have  to  devote 
more  attention  to  suitability  assessment.  Because  their 
training  and  experience  are  primarily  in  operational  and 
technical  issues,  the  DOT&E  staff s  focus  needs  to  be 
expanded  to  more  adequately  cover  suitability. 

The  inadequate  definition  of  key  suitability  issues  is  a 
major  shortcoming.  An  issue  may  be  critical  because  of 
risks  associated  with  new  technology  applications, 
operational  concepts,  or  support  concepts,  or  because  it 
represents  a  major  consumer  of  logistics  resources.  If  these 
risks  are  not  identified  early  and  addressed  in  test  plans, 
then  subsequent  testing  will  not  reveal  them. 

At  the  same  time,  it  is  not  clear  from  the  case  studies  that 
development  test  data  are  communicated  and  utilized  to  the 
extent  possible.  Nor  have  the  results  of  logistic  support 
analysis  (LSA)  and  the  integrated  logistics  support  (ILS) 
process,  where  critical  suitability  issues  are  identified,  been 
widely  or  actively  disseminated.  A  disciplined  acquisition 
process  should  strive  to  take  maximum  advantage  of  the 
already  available  information  to  help  make  suitability 
judgments. 
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Principles  of  Suitability  Assessment 

Placing  the  case-study  results  and  the  objective  of  the 
acquisition  process  in  this  context,  we  derive  four  principles 
for  assessing  the  suitability  of  major  systems: 

(1)  Assess  suitability  in  each  phase  of  development. 

(2)  Find  deficiencies  as  early  as  possible. 

(3)  Correct  deficiencies  in  the  most  appropriate 
phase. 

(4)  Coordinate  and  integrate  suitability  testing  and 
evaluation  across  all  phases. 

The  first  principle  follows  from  the  fact  that  opportunities  to 
assess  suitability  differ  at  each  phase  but  exist  in  all  phases. 
The  second  principle  is  simply  common  sense:  knowledge 
gained  early  can  inform  the  rest  of  the  process.  The  third 
principle  recognizes  that  although  a  problem  may  be 
known,  its  solution  may  best  be  pursued  at  a  later  date. 
There  is  no  need  to  stop  a  program  every  time  a  problem  is 
identified  until  it  is  fixed  -  particularly  because  suitability 
problems  can  often  be  surmounted  in  numerous  ways,  and 
which  way  is  best  may  not  be  immediately  evident.  This 
principle  should  not  be  taken  as  an  excuse  for  inaction  -  if 
a  problem  can  be  economically  solved  early,  and  most 
suitability  problems  cost  less  to  fix  early  than  late,  it  should 
be.  Finally,  it  is  clear  that  we  cannot  afford  to  perform 
separate  suitability  assessments  that  do  not  take  advantage 
of  information  already  gained.  It  is  not  businesslike  to  fail 
to  build  on  earlier  results  and  to  fail  to  think  ahead  to  the 
cost  and  suitability  drivers  for  fielded  systems. 

These  principles  do  not  require  new  policy  directives.  In 
fact,  adequate  policy  to  ensure  the  suitability  of  fielded 
systems  is  already  in  place.  The  current  5000  series  of  DoD 
instructions  and  its  predecessors  provide  the  direction  and 
define  the  appropriate  products  needed.  For  example,  the 
DOT&E  is  called  upon  to  assess  the  operational  suitability 
of  systems  before  they  enter  full-rate  production.  The 
DDDRE(T&E)  is  tasked  with  oversight  of  testing  for 
reliability,  and  the  Services’  development  testing  agencies 
are  tasked  with  certifying  that  a  system  is  ready  for 
operational  testing. 
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THE  NEED  TO  IMPROVE  IMPLEMENTATION 

If,  although  policy  is  adequate,  systems  are  being  fielded 
with  suitability  problems,  either  it  is  simply  impossible  to 
field  suitable  systems  or  else  policy  implementation  needs  to 
be  improved. 

The  case  studies  show  that  it  is  possible  to  field  suitable 
systems.  A  good  example  of  this  is  the  fielding  of  the  Army’s 
MSE.  The  system  (a  large  and  complex  communications 
system  supplying  the  telephone  network  -  including 
switches,  radio  linkages,  and  mobile  instruments  -  for  an 
Army  Corps)  was  suitable  when  fielded  and  performed 
beyond  expectation  in  Desert  Storm.  This  happened  in  part 
because  suitability  testing  discovered  problems  in 
reliability  and  training  that  were  overcome  by  a  CAP  prior 
to  fielding.  It  is  instructive  to  note  the  major  increases  in 
reliability  -  from  76  to  191  hours  between  operational 
mission  failures  for  the  node-center  switch,  for 
example  -  resulting  from  the  CAP  were  achieved  at  a  cost 
of  less  than  $10  million.  For  a  system  with  acquisition  cost 
in  excess  of  $4  billion,  this  was  a  small  sum.  It  is  clear, 
then,  that  suitable  systems  can  be  fielded,  that  an  active 
suitability  assessment  program  coupled  with  an  effective 
CAP  can  contribute  to  fielding  success,  and  that  the  cost  of 
doing  it  right  need  not  be  excessive. 

We  believe  that  policy  implementation  needs  to  be  improved 
for  more  suitable  systems  to  be  fielded.  In  the  remainder  of 
this  volume,  we  discuss  several  ways  in  which  this  might  be 
done. 
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CHAPTER  2.  RECOMMENDATIONS 


The  case  studies  show  that  critical  suitability  test  issues  are 
not  adequately  defined  and  followed  up  -  especially  issues 
primarily  associated  with  logistics  support.  Critical 
suitability  issues  drive  wartime  availability  and  peacetime 
costs.  To  use  scarce  test  and  evaluation  (T&E)  resources 
most  effectively,  the  T&E  community  needs  to  identify  the 
critical  suitability  issues  early  and  then  ensure  that  the 
T&E  plans  and  execution  are  consistent  with  the  key  issues. 

Current  policy,  as  embodied  in  the  5000  series  of 
instructions,  provides  for  thorough  treatment  of  suitability. 
In  this  chapter,  we  make  three  recommendations  for 
implementing  this  policy  better  and  one  recommendation  to 
strengthen  one  aspect  of  current  policy  -  testing  peculiar 
support  equipment. 

IMPRO  VED  IMPLEMENT  A  TION 

Improved  implementation  of  policy  is  necessary  to  better 
ensure  that  major  systems  are  suitable  when  fielded.  To 
accomplish  this,  suitability  assessments  should  be 
undertaken  according  to  the  four  principles  set  out  in  the 
preceding  chapter; 

(1)  Assess  suitability  in  each  phase  of  development. 

(2)  Find  deficiencies  as  early  as  possible. 

(3)  Correct  deficiencies  in  the  most  appropriate 
phase. 

(4)  Coordinate  and  integrate  suitability  testing  and 
evaluation  across  all  phases. 

Our  three  recommendations  to  put  these  principles  into 
practice  are: 

Recommendation  1.  Devote  increased  attention 
and  technical  support  to  suitability  assessment 
and  testing  at  all  phases  to  put  into  practice  the 
first  two  principles. 
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Chapter  3  suggests  how  suitability  assessments  could  be 
improved. 

This  first  recommendation  follows  from  our  observations 
from  the  case  studies  that  crucial  elements  of  suitability 
were  not  always  identified  as  such.  Increased  technical 
support  is  indicated  because  of  the  limited  size  and  scope  of 
expertise  of  DOT&E  staff  and  because  suitability  is  difficult 
to  assess.  Simply  stated,  there  is  no  “cookbook”  for 
evaluating  suitability.  For  example,  identifying  the  critical 
issues  for  evaluation  requires  knowledge  of  the  system 
itself,  its  expected  employment,  previous  systems,  the 
larger  logistics  system,  and  achievable  system  parameters. 
Because  of  the  complex  interactions  among  the  elements  of 
suitability  -  and  among  the  mission  equipment,  its 
operators  and  maintainers,  its  peculiar  support,  and  the 
larger  logistics  system  -  simple  rules  for  evaluating 
suitability  do  not  exist. 

The  complex  nature  of  suitability  assessment  means  that 
increased  levels  of  qualified  technical  support  will  be 
required  by  the  OSD  offices  to  identify  suitability  problems 
early  and  to  make  suitability  assessments  at  all  phases. 

Recommendation  2.  Place  increased  emphasis  on 
the  use  of  corrective  action  programs  as  follow¬ 
ups  to  suitability  assessments  to  improve  suit¬ 
ability.  This  will  put  into  practice  the  third 
principle,  correcting  problems  in  the  most 
appropriate  phase. 

The  case  studies  showed  the  successful  use  of  CAPs  for  two 
systems.  In  the  case  of  the  MSE,  for  example,  large 
improvements  to  system  suitability  were  achieved  at  low 
cost,  and  the  results  of  the  CAP  were  verified  by  follow-on 
operational  testing.  It  should  be  noted  that  the  MSE 
contract  included  performance  requirements  and 
incremental  purchases  that  motivated  improvement  by  the 
contractor.  The  planned  use  of  CAPs  recognizes  that  every 
system  is  likely  to  have  some  suitability  problems  during 
testing.  Being  prepared  to  correct  those  problems  through  a 
CAP  will  ensure  that  those  that  can  be  corrected  rapidly 
will  be,  and  that  those  whose  solution  may  need  to  be 
delayed  will  not  be  forgotten.  The  CAP  has  to  specify 
clearly  when  the  problems  will  be  corrected,  who  is 
responsible  for  making  the  correction,  and  where  the 
resources  to  make  the  correction  will  come  from.  The 
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informed  use  of  CAPs  will  help  overcome  suitability 
problems  in  the  appropriate  phase. 

Recommendation  3.  Make  results  and  data  from 
all  phases  and  disciplines  more  available  and 
encourage  their  use.  This  will  partially  accom¬ 
plish  the  fourth  principle,  more  coordination 
among  phases. 

The  case  study  results  showed  that  simply  taking 
advantage  of  existing  information  would  have  improved  the 
suitability  assessments  that  were  made.  As  a  first  step,  this 
information  should  be  made  readily  accessible  (in  fact,  it 
should  be  sent  to  the  offices  that  are  responsible  for 
suitability  assessment  as  a  matter  of  routine).  These  offices 
should  also  encourage  the  use  of  this  information  in 
assessments  they  perform  and  also  in  assessments 
analogous  Service  offices  perform.  Coordination  among  the 
OSD  offices  dealing  with  suitability  is  now  beginning  to  be 
pursued.  For  this  effort  to  succeed,  a  clear  picture  of  what 
information  needs  to  be  shared  should  be  developed. 

A  disciplined  process  for  testing  and  assessing  suitability, 
and  increased  resources  for  technical  support,  will  help 
ensure  that  suitability  issues  are  considered  and  proper 
actions  are  taken  to  get  more  suitable  systems  into  the  field. 
Judging  from  past  experience,  such  an  effort  is  needed  and 
will  pay  for  itself  many  times  over  in  improved  availability 
and  avoided  logistics  costs. 

PECULIAR  SUPPORT 

Peculiar  support  equipment  plays  a  crucial  role  in 
suitability  that  will  increase  as  systems  move  to  two-level 
maintenance.  Performance  of  the  critical  peculiar  support 
equipment  will  have  a  direct  and  significant  impact  on  the 
operational  suitability  of  a  weapon  system  in  both 
peacetime  and  wartime.  Therefore,  it  is  essential  that  the 
critical  support  equipment  be  part  of  the  total  operational 
suitability  assessment. 

If,  as  some  contend,  development  of  peculiar  support 
equipment  must  follow  that  of  the  end  item,  then  it  will  not 
be  possible  to  fully  assess  suitability  of  systems  at  OT&E. 
However,  if,  as  we  contend,  it  is  possible  to  develop  the 
peculiar  support  concurrently,  then  crucial  items  can  and 
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should  be  made  available  for  test.  We  make  the  following 
recommendation  to  improve  peculiar  support: 

Recommendation  4.  Require  critical  peculiar 
support  equipment  to  be  made  available  for 
initial  OT&E. 

The  remainder  of  this  volume  illustrates  how  these 
recommendations  might  be  applied  and  how  they  could  be 
put  into  practice  by  DOT&E.  A  similar  analysis  should  be 
performed  for  the  other  offices  involved  in  suitability 
assessments  [such  as,  at  the  OSD  level,  DDDRE(T&E)]. 
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CHAPTER  3 .  IMPROVING  SUITABILITY 
ASSESSMENTS 


DOT&E  is  positioned  to  motivate  improved  treatment  of 
suitability  in  OT&E.  For  the  Milestone  III  decision, 
DOT&E  reports  to  the  Secretary  of  Defense  and  Congress  on 
the  adequacy  of  the  operational  test  and  on  whether  or  not 
the  system  is  suitable.  In  addition,  DOT&E  has  signature 
approval  of  the  Test  and  Evaluation  Master  Plan  (TEMP) 
and  operational  test  plans. 

These  opportunities  to  influence  the  treatment  of 
operational  suitability  for  major  acquisition  programs  are 
highlighted  in  Figure  3-1,  extracted  from  DoDI  5000.2. 
Each  box  with  a  darkened  border  represents  a  step  in  which 
DOT&E  has  a  review  and  approval  role. 

To  exercise  this  authority,  DOT&E  needs  to  ensure  that  the 
TEMP  and  test  plans  provide  for  adequate  treatment  of 
suitability.  This  chapter  describes  some  candidate  tools 
that  can  be  applied  by  DOT&E  action  officers  to  provide  this 
assurance. 

Suitability  depends  on  many  players,  including  the  project 
manager,  the  intended  user,  the  supporting  commands, 
Service  testing  agencies,  DDDRE(T&E),  and  DOT&E.  The 
suggestions  in  this  chapter  were  originally  conceived  for  use 
by  DOT&E,  but  they  could  be  useful  to  the  other  players  as 
well. 

METHOD  FOR  SUITABILITY  ASSESSMENT 

We  have  identified  four  primary  actions  that  DOT&E  can 
apply  throughout  the  development  process  to  implement  the 
recommendations  from  Chapter  2.  They  are  shown  in 
Figure  3-2. 

These  four  actions  are  included  in  DOT&E’s  current 
approach.  However,  as  evidenced  by  the  case  studies  (and  a 
brief  review  of  two  systems  now  in  development),  these 
actions  are  not  adequately  performed. 
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FIG.  3- 1.  TESTERS'  PROCESS  FLOW  IN  WEAPON  SYSTEMS  DEVELOPMENT 
UNDER  DoD  DEFENSE  ACQUISITION  POLICY 
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FIG.  3-2.  PRIMARY  DOT&E  ACTIONS  FOR  IMPROVING  THE  TREATMENT 
OF  OPERATIONAL  SUITABILITY 

The  first  three  of  these  actions  should  be  performed  at  least 
once  in  each  major  acquisition  (and  may  need  to  be  repeated 
in  response  to  changes  in  mission  requirements,  operations 
and  support  concept,  or  design).  If  the  system  fully  satisfies 
all  of  its  effectiveness  and  suitability  requirements  at 
OT&E,  then  a  corrective  action  plan  may  not  be  needed.  (As 
a  practical  matter,  major  weapon  systems  and  their 
acquisitions  are  sufficiently  complex  that  some  corrections 
will  almost  certainly  be  required  after  initial  OT&E.) 
Specific  suggestions  for  accomplishing  the  primary  actions 
are  presented  in  the  following  sections. 

Identify  Key  Suitability  Issues 

Key  suitability  issues  are  those  aspects  of  suitability  that 
could  cause  system  performance  or  cost  to  be  unacceptable. 
They  are  either  areas  of  risk  or  logistics  drivers  (i.e.,  major 
consumers  of  logistics  resources). 

Typically,  suitability  risks  are  associated  with  major 
changes  in  one  or  more  of  the  following  areas: 

•  Technology  and  its  applications 

•  Operational  concept 

•  Support  concept. 

For  example,  the  AH-64’s  composite  material  main  rotor 
blade  represented  the  first  use  of  that  material  in  such  a 
demanding  application.  The  originally  expected 
replacement  interval  was  1,500  flight  hours.  In  contrast, 
field  data  indicate  a  replacement  interval  of  only  164  hours. 
For  examples  of  changes  in  the  support  concept,  consider  the 
Advanced  Tactical  Fighter  (ATF)  and  the  RAH-66 
Comanche  helicopter.  Both  vehicles  plan  to  use  two-level 
maintenance,  a  major  change  from  the  traditional 
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three-level  approach.  Because  success  of  the  two-level 
concept  is  critically  dependent  on  the  accuracy  of  the  fault 
diagnosis  system,  fault  diagnosis  is  a  key  suitability  issue. 

Logistics  drivers  are  the  system  characteristics  that 
consume  the  majority  of  the  logistics  resources.  Examples 
of  logistics  drivers  are  high-failure-rate  modules,  inaccurate 
fault  diagnostics,  and  mismatches  between  maintenance 
skills  and  requirements.  The  logistics  drivers  may  or  may 
not  be  associated  with  logistics  risks.  Even  though  a 
logistics  driver  may  be  of  low  risk,  it  should  be  examined 
because  of  its  importance  to  affordable  system  performance. 

Table  3-1  displays  three  methods  for  identifying  key 
suitability  issues.  They  are  presented  in  order  of  increasing 
resource  requirements. 

TABLE  3-1 

METHODS  FOR  IDENTIFYING  KEY  SUITABILITY  ISSUES 


Method  1 

Method  2 

Method  3 

Document  review 

Analysis 

Simulation 

List  suitability  issues  and 

Compare  with  other 

Construct  a  model  of 

goals  from  system 

systems 

system  costs  and 

requirements  documents 

Review  the  elements  of 

performance;  include 
operations  and  support 

Check  for  issues  and  goals 

integrated  logistics 

concepts 

in  TEMP 

support 

Analyze  the  impact  of 

Analyze  the  relationships 

changes  in  operations  and 

among  the  measures  and 
assumptions  used 

support  variables 

Document  Review 

The  document  review  method  requires  the  least  time  and 
effort.  It  is  suited  to  finding  large  inconsistencies  between 
requirements  documents  and  the  TEMP. 

Table  3-2  lists  relevant  requirements  documents.  They 
include  high-level  descriptions  of  the  operations  and 
support  concepts.  These  documents  provide  explicit  and 
implicit  descriptions  of  suitability  issues.  The  document 


Ch  ter  3.  Improving  Suitability  Assessments 


Page  20 


review  method  does  not  ensure  that  all  the  suitability  issues 
are  addressed  (since  the  requirements  documents  may  not 
provide  a  comprehensive  treatment  of  suitability  issues). 
But  since  it  is  fast  and  easy  to  apply,  it  offers  significant 
benefits  for  the  effort  required. 

TABLE  3-2 

SYSTEM  DOCUMENTS  CONTAINING  SUITABILITY  REQUIREMENTS 


Document 

Program  phase 

Mission  need  statement 

Operational  requirements  document 
Logistic  support  analysis  results 

RAM  rationale  report 

Milestone  0 

Milestones  1  and  II 

Milestones  1,  II,  and  III 

Milestones  1  and  II 

Note:  RAM  =  Reliability,  Availability,  Maintainability 


Perhaps  the  best  source  of  suitability  issues  (at  least 
theoretically)  is  th-  ’  oumentation  of  the  selected  tasks 
from  logistic  support  analysis  (LSA).  These  tasks  are 
describ  ed  in  MIL-STD-1388-1A.  Their  purpose  is  to  identify 
the  critical  support  issues. 


Selected  Logistic  Support  Analysis  (LSA)  Tasks 

Task  Section  200:  Mission  and  Support  Systems  Definition 

Purpose:  To  establish  supportability  objectives  and  supportability-related  design  goals, 
thresholds,  and  constraints  through  comparison  with  existing  systems  and  analysis  of 
supportability,  cost,  and  readiness  drivers. 

Task  301:  Functional  Requirements  Identification 

Purpose:  To  identify  the  operations  and  support  functions  that  must  be  performed  for 
each  system  alternative  under  consideration  and  then  identify  the  tasks  that  must  be  performed 
in  order  to  operate  and  maintain  the  new  system  in  its  intended  environment. 

Task  303:  Evaluation  of  Alternatives  and  Tradeoff  Analysis 

Purpose:  To  determine  the  preferred  support  system  alternative(s)  for  each  weapon 
system  alternative  and  to  participate  in  alternative  system  tradeoffs  to  determine  the  best 
approach  for  satisfying  the  need  with  the  best  balance  among  cost,  schedule,  performance, 
readiness,  and  supportability. 
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Both  DoDI  5000.2  and  MIL-STD-1388-1A  require  LSA  to  be 
performed  at  each  stage  of  development,  with  the  level  of 
detail  increasing  as  system  development  proceeds.  “The 
logistics  support  analysis  process  will  be  used  to:  (1)  develop 
and  define  supportability  related  design  factors,  and  (2) 
ensure  the  development  of  a  fully  integrated  system  support 
structure.”  (DoDI  5000.2,  Part  7,  Section  A,  p.  7-A-3)  The 
program  office  is  responsible  for  the  LSA.  For  most  major 
systems,  the  LSA  is  performed  by  contractors.  The  LSA 
reports  can  be  used  by  DOT&E  to  identify  the  key 
suitability  issues. 


Example  -  Document  Review 

We  applied  the  "document  review"  method  to  a  major  aeronautical  system  currently  in  concept 
development.  A  brief  review  indicated  the  following  key  suitability  issues:  operational 
availability  (A„),  mean  time  between  mission  failure  (MTBMF),  direct  maintenance  man-hours 
per  flight  hour  (DMMH/FH),  transportability,  and  two-level  maintenance.  All  but  the  last  issue 
have  quantitative  criteria. 


There  is  no  guarantee,  however,  that  the  suitability  issues 
derived  from  the  system  documents  will  be  comprehensive. 
DOT&E  needs  a  comprehensive  list  of  the  key  suitability 
issues. 

Analysis 

A  more  involved  and  more  thorough  approach  for 
identifying  the  key  suitability  issues  is  analysis.  In  this 
context,  analysis  is  the  assessment  of  the  suitability 
characteristics  and  resources  required  by  the  proposed 
design  to  operate  in  the  expected  scenarios.  Analysis  may 
be  used  to  determine  when  the  key  suitability  issues  are 
comprehensive,  consistent,  and  credible. 

Two  sources  of  candidate  suitability  issues  for  consideration 
are  the  DOT&E  Operational  Suitability  Guide,  Volume  I  — 
A  Tutorial,  (February  1990),  and  the  elements  of  integrated 
logistic  support. 

Analysis  can  be  performed  by  comparing  the  system  being 
assessed  with  similar  systems  (or  subsystems)  already 
fielded.  This  approach  requires  consideration  of  the 
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suitability  and  logistics  support  elements  in  the  context  of 
the  mission  and  the  operations  and  support  concepts  for  the 
proposed  system.  It  requires  familiarity  with  the  elements 
of  logistics  and  how  they  are  used  in  various  support 
concepts.  It  benefits  from  the  analyst’s  experience  with 
other  weapon  systems. 

Another  analysis  approach  is  to  review  the  logical 
relationships  among  the  quantitative  measures  of 
suitability  and  to  determine  the  assumptions  on  which  the 
measures  are  based.  By  perturbing  the  assumed  values,  the 
analyst  can  assess  the  significance  of  the  effect  of  assumed 
values  on  achieving  the  stated  quantitative  goals.  If 
attainment  of  a  goal  is  sensitive  to  an  assumed  value  for 
some  parameter,  then  that  parameter  may  be  a  candidate 
for  OT&E. 


Example  -  Sensitivity  Analysis 

For  the  aforementioned  aeronautical  system,  A„  was  found  to  be  a  function  of  the  mean  time 
between  mission  failure  (MTBMF),  the  mean  time  to  restore  (MTTR)  the  system, and  the 
administrative  and  logistics  down  time  (ALDT)  per  maintenance  action.  Goals  were  provided 
for  MTBMF  and  MTTR.  The  ALDT  value  was  derived  from  a  model  of  the  aircraft 
maintenance  process,  but  no  rationale  for  the  assumed  values  was  provided  in  the  available 
documents.  Based  on  experience,  the  assumed  ALDT  value  was  judged  to  be  optimistic. 
Furthermore,  a  1-hour  increase  in  ALDT  requires  an  increase  of  over  1  hour  in  the  MTBMF 
goal  for  the  system  to  achieve  its  A„  goal.  This  finding  suggests  that  the  ALDT  should  be 
assessed  in  OT&E. 


Analysis  requires  more  effort  than  document  review,  but  it 
is  more  likely  to  result  in  a  list  of  key  suitability  issues  that 
will  be  necessary  and  sufficient  to  determine  the  system’s 
suitability  at  Milestone  HI. 

Simulation 

The  third  and  most  involved  method  is  simulation.  In  this 
method,  models  are  constructed  to  represent  the  cost  and 
performance  of  the  proposed  design  in  the  expected 
scenarios.  Cost  and  performance  should  be  expressed  as 
functions  of  suitability  elements.  The  models  are  then 
exercised  with  varying  sets  of  input  values  to  estimate  the 
impacts  of  changes  in  suitability  characteristics.  Modeling 
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and  analysis  encourage  the  development  of  an  explicit 
rationale  to  explain  why  certain  suitability  issues  are  key. 
However,  the  effort  required  to  develop  and  certify  a  model 
may  be  extensive. 

In  some  cases,  it  may  be  possible  to  use  existing  models.  In 
others,  it  may  be  necessary  to  construct  an  appropriate 
model. 


Example  -  Simulation 

For  the  aeronautical  system  in  the  preceding  examples,  one  could  build  a  computer  model  to 
simulate  one  squadron  operating  according  to  the  operatinal  requirements  document  under  the 
assumed  manning  levels,  failure  rates,  maintenance  times,  fault  diagnostics  performance,  and 
depot  support.  The  simulation  could  be  used  to  estimate  the  major  consumers  of  logistics 
resources.  The  assumed  inputs  could  be  perturbed  to  identify  the  assumptions  that  could  have 
the  most  severe  impact  on  the  ability  of  the  system  to  satisfy  its  top-level  performance 
requirements.  The  results  would  indicate  key  suitability  issues. 


Review  OT&E  Plans 

Once  the  key  suitability  issues  have  been  identified,  the 
next  step  is  to  ensure  that  the  OT&E  plans  address  the 
issues.  The  fundamental  question  is  whether  or  not 
execution  of  the  OT&E  plans  will  provide  an  adequate  basis 
for  assessing  the  key  suitability  issues. 

Some  suitability  issues  have  well-defined  measures  that 
depend  only  on  the  prime  mission  equipment.  Other  issues 
are  less  well  defined  and  depend  on  interactions  with  other 
systems  or  resources.  The  test  planners  must  decide  how 
each  suitability  measure  will  be  addressed,  whether  it  be  by 
testing,  modeling,  expert  judgment,  or  other  means. 
DOT&E  must  decide  whether  the  treatment  given  to  each 
suitability  issue  is  appropriate. 

Several  methods  of  review  are  possible.  The  simplest 
method  is  a  mechanical  check  using  the  following  questions: 

•  Is  each  issue  addressed  in  the  plan? 

•  Are  test  resources  identified? 
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•  Are  test  personnel  identified? 

•  Are  there  provisions  for  collection  of  test  data? 

•  Are  models  to  be  used?  Are  they  identified  and 
justified? 


Example  -  TEMP  Review 

The  OT&E  chapter  of  the  TEMP  for  the  aeronautical  system  does  not  address  transportability, 
which  was  identified  as  a  key  suitability  issue  in  the  requirements  documents.  In  addition,  no 
attention  is  given  to  verifying  the  ALDT  estimate. 


Answers  to  the  preceding  questions  are  necessary  to 
determine  the  adequacy  of  OT&E  plans,  but  they  may  not 
be  sufficient.  Analysis  of  the  test  duration  and  conditions 
may  be  required. 

For  measures  that  are  to  be  tested,  the  nature  and  length  of 
testing  should  be  sufficient  to  estimate  the  parameter  with 
reasonable  confidence.  Some  factors  to  consider  are: 

•  Treatment  of  scheduled/phase  maintenance 

•  Treatment  of  inspection  time 

•  Representative  faults  for  the  fault  diagnosis 
system  and  maintenance  actions 

•  Representative  environments  and  stresses. 

Table  3-3  shows  one  format  (with  hypothetical  entries)  that 
could  be  used  for  checking  the  coverage  of  suitability  issues 
by  the  TEMP.  Such  a  table  provides  a  mechanism  to  ensure 
that  test  plans  are  thoroughly  reviewed. 

Some  measures  may  be  evaluated  by  using  models.  Because 
the  complete  logistic  support  structure  will  rarely  be 
available  prior  to  Milestone  III,  OT&E  cannot  provide  a 
thorough  test  of  suitability  issues  that  depend  on  the 
complete  structure.  Availability  is  one  such  issue.  Depot 
support  may  be  another  one. 

DOT&E  should  consider  the  credibility  of  the  proposed 
models.  The  model  may  have  been  certified  for  application 
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TABLE  3-3 

EXAMPLE  FOR  TRACKING  TREATMENT  OF  KEY  SUITABILITY  ISSUES  IN  THE  TEMP 

(The  entries  are  hypothetical) 


Issue 

Measure 

TEMP  coverage 

Comments 

Resources 

Scenarios 

Data 

Availability 

A0  in  combat 
scenario 

Limited 

Adequate 

Adequate 

Use  model  to 
combine  R&M 
and  support 
data 

Reliability 

MTBCF 

Adequate 

Limited 

Adequate 

Use  models  to 

augment 

scenarios 

Mobility 

Pack/unpack 

times 

Adequate 

Adequate 

Adequate 

Physical  fits 
verified  in  DT 

Depot  support 

TBD 

No 

No 

No 

Note:  MTBCF  =  mean  time  between  critical  failure;  TBD  =  to  be  determined,  R&M  =  reliability  and  maintainability 


to  the  current  problem.  If  not,  then  it  should  have  been 
subjected  to  a  general  verification  and  validation  process.  If 
neither  of  these  reviews  have  been  accomplished,  then 
DOT&E  should  consider  requiring  the  test  plan  to  include 
some  assessment  of  the  model.  DOT&E  should  consider  the 
assumptions  used  by  the  moael.  (Examples  include  ALDT, 
order  and  shipping  times,  and  availability  of  spares  and 
consumables.)  If  the  model  is  sensitive  to  the  assumptions 
and  the  assumptions  are  not  well  justified,  then  DOT&E 
should  consider  including  the  assumptions  in  the  test  and 
evaluation  plans. 

Perform  OT&E  Assessments 

After  the  operational  test  is  performed,  DOT&E  reviews  the 
results  and  develops  its  own  interpretation  of  the  results. 
The  key  suitability  issues  need  to  be  assessed  along  with  the 
effectiveness  issues.  The  assessment  should  recognize  the 
suitability  issues  that  could  not  be  fully  assessed  in  OT&E, 
the  attendant  risks,  and  the  need  for  any  follow-on  tests  to 
verify  suitability. 
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Questions  such  as  the  following  should  be  addressed: 

•  Is  the  assessment  comprehensive  with  respect  to 
the  key  suitability  issues? 

•  Are  the  major  assumptions  justified? 

•  Are  there  suitability  problems  that  should  receive 
corrective  action? 

•  Are  there  key  suitability  issues  that  require 
further  OT&E? 

OT&E  assessments  are  limited  by  the  scope  of  OT&E. 
Practical  constraints  on  test  time,  maturity  of  peculiar 
support  elements,  and  interfaces  with  the  general  logistics 
support  structure  restrict  the  ability  to  fully  demonstrate 
operational  suitability.  It  may  be  possible  to  mitigate  these 
restrictions  by  augmenting  the  test  results  with  analysis, 
data  from  other  sources,  and  review  of  the  plans  for  the 
affected  suitability  elements. 

The  results  of  OT&E  may  have  utility  beyond  the 
Milestone  HI  decision.  Potential  beneficiaries  of  the  OT&E 
results  include  the  Cost  and  Operational  Effectiveness 
Analysis  (COEA),  logistic  support  analysis  and  integrated 
logistic  support  plans  (LSA/ILSP),  development  testing,  and 
operating  doctrine. 

Contribute  to  Corrective  Action  Plans 

Because  of  the  complexity  of  modern  weapon  systems,  it  is 
likely  that  not  all  suitability  issues  can  be  fully  resolved 
during  OT&E  and  that  some  modifications  to  the  system 
will  be  made  as  a  result  of  OT&E.  DOT&E  could  note  in 
their  report  whether  a  well-defined  CAP  is  in  place  to  make 
these  modifications,  whether  it  is  resourced,  and  whether 
responsibilities  have  been  clearly  assigned.  DOT&E  should 
also  review  the  status  of  the  suitability  assessment  and  the 
scope  of  any  proposed  design  changes  to  determine  the  need 
for  FOT&E. 

SUMMARY 

DOT&E  has  been  given  a  major  statutory  role  in  assessing 
and  reporting  on  operational  effectiveness  and  suitability. 
The  steps  outlined  above  should  help  DOT&E  fulfill  that 
role  more  efficiently  and  effectively  and  thus  help  ensure 


the  fielding  of  suitable  systems.  Similar  steps  by  the 
development  testing  community  could  further  enhance 
system  suitability. 
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GLOSSARY 


Ao 

= 

operational  availability 

ALDT 

= 

administrative  and  logistics  down  time 

ATF 

= 

Advanced  Tactical  Fighter 

BIT 

= 

built-in  test 

B-LRIP 

= 

beyond  low-rate  initial  production 

CAP 

— 

corrective  action  program 

COEA 

= 

Cost  and  Operational  Effectiveness 
Analysis 

DDDRE(T&E) 

— 

Deputy  Director  of  Defense  Research 
and  Engineering  (Test  and  Evaluation) 

DMMH/FH 

— 

direct  maintenance  man-hours  per 
flight  hour 

DOT&E 

= 

Director,  Operational  Test  and 
Evaluation 

DT 

— 

development  testing 

DT&E 

— 

development,  test,  and  evaluation 

DUSD(A) 

= 

Deputy  Under  Secretary  of  Defense 
(Acquisition) 

EETF 

— 

electronic  equipment  test  facility 

FD/LS 

— 

Fault  Detection/Location  System 

FOT&E 

= 

follow-on  operational  test  and 
evaluation 

ILS 

= 

integrated  logistic  support 

LCAC 

Landing  Craft  Air  Cushion 

LRU 

line  replaceable  unit 
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LSA 

— 

logistic  support  analysis 

LSA/ILSP 

— 

logistic  support  analysis  and 
integrated  logistic  support  plans 

MSE 

= 

mobile  subscriber  equipment 

MTBCF 

— 

mean  time  between  critical  failure 

MTBMF 

— 

mean  time  between  mission  failure 

MTTR 

= 

mean  time  to  restore 

OSD 

— 

Office  of  the  Secretary  of  Defense 

OT 

= 

operational  testing 

OT&E 

= 

operational  test  and  evaluation 

RAM 

— 

reliability,  availability, 
maintainability 

R&M 

— 

reliability  and  maintainability 

PM 

— 

program  manager 

T&E 

— 

test  and  evaluation 

TBD 

— 

to  be  determined 

TEMP 

Test  and  Evaluation  Master  Plan 
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APPENDIX.  U.S.  CODE,  TITLE  10,  SECTION 2399 


<£2399.  Operational  test  and  evaluation  of  defense  acquisition 
programs 

(a)  Condition  for  proceeding  beyond  low-rate  initial  production. 

(1) The  Secretary  of  Defense  shall  provide  that  a  major  defense 
acquisition  program  may  not  proceed  beyond  low-rate  initial 
production  until  initial  operational  test  and  evaluation  of  the  program 
is  completed.  (2)  In  this  subsection,  the  term  “major  defense 
acquisition  program”  means- 

(A)  a  conventional  weapons  system  that  is  a  major  system  within 
the  meaning  of  that  term  in  section  2302(5)  of  this  title;  and 

(B)  is  designed  for  use  in  combat. 

(b)  Operational  test  and  evaluation.  (1)  Operational  testing  of  a  major 
defense  acquisition  program  may  not  be  conducted  until  the  Director  of 
Operational  Test  and  Evaluation  of  the  Department  of  Defense 
approves  (in  writing)  the  adequacy  of  the  plans  (including  the 
projected  level  of  funding)  for  operational  test  and  evaluation  to  be 
conducted  in  connection  with  that  program. 

(2)  The  Director  shall  analyze  the  results  of  the  operational  test  and 
evaluation  conducted  for  each  major  defense  acquisition  program.  At 
the  conclusion  of  such  testing,  the  Director  shall  prepare  a  report 
stating  the  opinion  of  the  Director  as  to- 

(A)  whether  the  test  and  evaluation  performed  were  adequate;  and 

(B)  whether  the  results  of  such  test  and  evaluation  confirm  that 
the  items  or  components  actually  tested  are  effective  and  suitable 
for  combat. 

(3)  The  Director  shall  submit  each  report  under  paragraph  (2)  to  the 
Secretary  of  Defense,  the  Under  Secretary  of  Defense  for  Acquisition, 
and  the  congressional  defense  committees.  Each  such  report  shall  be 
submitted  to  those  committees  in  precisely  the  same  form  and  with 
precisely  the  same  content  as  the  report  originally  was  submitted  to 
the  Secretary  and  Under  Secretary  and  shall  be  accompanied  by  such 
comments  as  the  Secretary  may  wish  to  make  on  the  report. 

(4)  A  final  decision  within  the  Department  of  Defense  to  proceed  with 
a  major  defense  acquisition  program  beyond  low-rate  initial 
production  may  not  be  made  until  the  Director  has  submitted  to  the 
Secretary  of  Defense  the  report  with  respect  to  that  program  under 
paragraph  (2)  and  the  congressional  defense  committees  have  received 
that  report. 

(5)  In  this  subsection,  the  term  “major  defense  acquisition  program" 
has  the  meaning  given  that  term  in  section  138(a)(2)(B)  of  this  title. 

(c)  Determination  of  quantity  of  articles  required  for  operational 
testing.  The  quantity  of  articles  of  a  new  system  that  are  to  be  procured  for 
operational  testing  shall  be  determined  by- 

(1)  the  Director  of  Operational  Test  and  Evaluation  of  the  Department 
of  Defense,  in  the  case  of  a  new  system  that  is  a  major  defense 
acquisition  program  (as  defined  in  section  138(a)(2)(B)  of  this  title);  or 
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(2)  the  operational  test  and  evaluation  agency  of  the  military 
department  concerned,  in  the  case  of  a  new  system  that  is  not  a  major 
defense  acquisition  program. 

(d)  Impartiality  of  contractor  testing  personnel.  In  the  case  of  a  major 
defense  acquisition  program  (as  defined  in  subsection  (a)(2)),  no  person 
employed  by  the  contractor  for  the  system  being  tested  may  be  involved  in 
the  conduct  of  the  operational  test  and  evaluation  required  under 
subsection  (a).  The  limitation  in  the  preceding  sentence  does  not  apply  to 
the  extent  that  the  Secretary  of  Defense  plans  for  persons  employed  by 
that  contractor  to  be  involved  in  the  operation,  maintenance,  and  support 
of  the  system  being  tested  when  the  system  is  deployed  in  combat. 

(e)  Impartial  contracted  advisory  and  assistance  services.  (1)  The 
Director  may  not  contract  with  any  person  for  advisory  and  assistance 
services  with  regard  to  the  test  and  evaluation  of  a  system  if  that 
person  participated  in  (or  is  participating  in)  the  development, 
production,  or  testing  of  such  system  for  a  military  department  or 
Defense  Agency  (or  for  another  contractor  of  the  Department  of 
Defense). 

(2)  The  Director  may  waive  the  limitation  under  paragraph  (1)  in  any 
case  if  the  Director  determines  in  writing  that  sufficient  steps  have 
been  taken  to  ensure  the  impartiality  of  the  contractor  in  providing  the 
services.  The  Inspector  General  of  the  Department  of  Defense  shall 
review  each  such  waiver  and  shall  include  in  the  Inspector  General’s 
semi-annual  report  an  assessment  of  those  waivers  made  since  the  last 
such  report. 

(3)  A  contractor  that  has  participated  in  (or  is  participating  in)  the 
development,  production,  or  testing  of  a  system  for  a  military 
department  or  Defense  Agency  (or  for  another  contractor  of  the 
Department  of  Defense)  may  not  be  involved  (in  any  way)  in  the 
establishment  of  criteria  for  data  collection  performance  assessment, 
or  evaluation  activities  for  the  operational  test  and  evaluation. 

(f)  Source  of  funds  for  testing.  The  costs  for  all  tests  required  under 
subsection  (a)  shall  be  paid  from  funds  available  for  the  system  being 
tested. 

(g)  Director’s  annual  report.  As  part  of  the  annual  report  of  the 
Director  under  section  138  of  this  title,  the  Director  shall  describe  for  each 
program  covered  in  the  report  the  status  of  test  and  evaluation  activities  in 
comparison  with  the  test  and  evaluation  master  plan  for  that  program,  as 
approved  by  the  Director.  The  Director  shall  include  in  such  annual  report 
a  description  of  each  waiver  granted  under  subsection  (e)(2)  since  the  last 
such  report. 

(h)  Definitions.  In  this  section: 

(1)  The  term  “operational  test  and  evaluation”  has  the  meaning  given 
that  term  in  section  138(a)(2)(A)  of  this  title  For  purposes  of 
subsection  (a),  that  term  does  not  include  an  operational  assessment 
based  exclusively  on- 

(A)  computer  modeling; 

(B)  simulation;  or 

(C)  an  analysis  of  system  requirements,  engineering  proposals, 
design  specifications,  or  any  other  information  contained  in 
program  documents. 

(2)  The  term  “congressional  defense  committees"  means  the 
Committees  on  Armed  Services  and  the  Committees  on  Appropriations 
of  the  Senate  and  House  of  Representatives. 


